Measuring in design

AI-assisted summary

Measuring in design follows the fundamental principle that designers should pay attention to what users do, not what they say. Effective measurement requires watching users as they attempt to perform tasks with the user interface, and collecting preference data only after users have actually used a design and have real experience with it.

The field encompasses various metric types including performance metrics (task completion, time, error rates), user problem related metrics (accessibility, navigation issues), self-reporting metrics (satisfaction scales), and behavioral/physiological metrics (eye-tracking, emotion measurement). The HEART framework provides guidance for selecting appropriate metrics, and combining performance with self-reported metrics typically yields the best results for understanding user experience.

To design the best UX, pay attention to what users do, not what they say
To discover which designs work best, watch users as they attempt to perform tasks with the user interface
When should you collect preference data from users? Only after they have used a design and have a real feeling for how well it supports them

Key concepts

Metrics – verified indicators for measuring attributes or aspects
- UX Metriky & CX Metrics
Frameworks – verified procedures and guides
- e.g. B-COM model
Theory – verified descriptions of reality and their explanations
- e.g. Principle of least effort

Research operationalization

I want to learn something -> Which metrics should I use -> How do I measure it?

Usually measuring

Objective
- performance
Subjective
- (do not) like, perceived effort, satisfaction

Types of studies

Transation/task completion
Product comparison
Navigation and information architecture
Usability and usefulness
Alternative design comparison

Metric types

Performance metrics
- quantitative
- task/transaction completion
  - yes/no or possible finer grain (full, partial success, ...)
- time spent on a task
- error rate
- efficiency
  - steps needed to reach a goal
- adoption / learning curve / learnability after repeated usage
  - important for tools with repetetive usage
User problem related metrics
- e.g. wayfinding in exterior
  - missing signage
  - tilted road
  - decreased visibility and readability
  - no guidance for those with special needs
- e.g. web accessibility
  - unavailable information
  - malfunctioning search
  - incomplete breadcrumbs navigation
  - low contrast and readability
Self-reporting metrics
- all sorts of scales
  - Lickert, semantic differential (agree / disagree) , TAM, SUS, UMUX, UMUX-lite
Behavioral and physiological metrics
- eye-tracking, emotion measurement, stress measurement

Tips

combine performance and self-reported metrics for best results
HEART framework could be helpful when deciding for which metrics to use on a project